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What People Really Care About 

When the AAHE (American Association for Higher Education) Assessment Forum developed the 9 
Principles of Good Practice for Assessing Student Learning, Principle 7 stated that assessment makes a difference 
when it illuminates questions that people really care about” (Astin). The first principle recognizes that 
assessment “is not an end in itself but a vehicle for educational improvement.” This paper discusses how our 
department curricula, classroom environments, and ultimately student learning might be improved by “high tech” 
assessment and evaluation techniques to find out “what people really care about”. Collaborative technologies are 
used to gather and process the opinions of students, faculty, and other stakeholders. 

Five years ago, the college president challenged the faculty to examine how students learn and what each 
department’s curriculum should look like for the 21 st Century. He strongly encouraged us to begin active planning 
for the continued intellectual vibrancy of the College and its curriculum. Because “students learn more when there is 
a match between their abilities and the curriculum” (Ratcliff, 1995), to remain intellectually vibrant demands that 
our college examine regularly what we are trying to accomplish as well as what is contributing to student success 
and what is not. Several groups met regularly to discuss curricular changes: Faculty Council, Administrative 
Council, the President’s Council, Teaching Groups, Academic Affairs Committee, and fifty senior students. The use 
of collaborative technologies allowed the following: gathering of divergent opinions at times convenient to 
participants; meetings dominated by content, not personality; adherence to a structured agenda, which resulted in 
consistency in issues discussed without loss of those ideas after the session ends. Throughout the sequence of those 
meetings, there was convergence of the following agreements: we validated a common core set of required courses 
for all students, but called for rethinking its content and staffing; we challenged the structure of the current 
distribution system; and we validated the need for more intentional inclusion of writing throughout the curriculum. 
The outcome of the meetings was a mandate for forming a curriculum review team, which recently finished its 
work. 

Educational institutions nationwide are expected to be increasingly accountable for the attainment of the 
stated goals in the form of demonstrable changes in students. Curriculum is one part of the total institutional 
improvement which, in combination with other qualities, has an important impact on student development. It seems 
imperative, then, to include our students in the process of evaluating the curriculum and the value faculty members 
add to their educational process. Using the collaborative technologies as a means for collecting these opinions 
provides a safe and exciting forum for discussion of issues important to the students. Table 1 summarizes how six 
departments used the collaborative facilities for assessment purposes. 



Education 


To assess the teaching skills needed by majors 


Music 


To assess the information technology needs for music courses 


Computer Science/MIS 


To assess the adequacy of computer lab support for departmental courses 


History 


To assess the adequacy of history curriculum in meeting certification 
requirements for teacher licensure 


Chemistry 


To discuss the changes needed, problems of, teaching of, and the 
contents of the general introductory chemistry course. 


Freshman English (Paideia) program 


To discuss and plan the curriculum for the 16 th Century unit of the course 



E- Listening and Collaborative Technologies 

We define “E- Listening” as the use of collaborative technologies to gather and process the opinions of 
students and other stakeholders for departmental improvement, and the attempt to extend the process and its benefits 
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to the broader institution. Collaborative technologies are broadly defined as those that enable collaboration among 
individuals engaged in a common task (Kock, 2000). In this case, the common task is assessment for departmental 
improvement. 

This process of improvement depends on continuous feedback with appropriate response. It begins with the 
collection, organization, analysis and reporting of student opinions and assessment data. The authors were charged 
with designing and implementing our departmental assessment programs. These included all the usual tasks of 
writing of mission statements, identifying goals and objectives, and developing means to assess whether we had 
accomplished what we hoped. We represent two different undergraduate departments (Department of Education and 
Department of Computer Science) which share the requirement that they must solicit, organize, analyze and report 
on the annual collections of graduating senior exit interviews. The tasks are cumulative, in that prior information is 
compared with the most recent collection. The enormous amount of data proves cumbersome to record, analyze and 
store. 

The Software Tools 

Imagine a tool that allows the structuring of assessment or evaluation questions and activities, the capturing 
of important ideas, prioritizing issues and opinions, instant reporting of results, use of an outline tool to write 
planning documents based on the data collected, and tools to create reports on any part or all of the data collected: 
that’s what collaboration tools can do! 

We designed and coordinated assessment activities using two different collaborative technologies, Group 
Systems and Facilitate.com. In Figure 1, students are shown in the Round Table Room at Luther College, using 
software tools such as Brainstormer, Topic Commenter, and Categorizer, to offer their opinions during such 
activities as the senior exit interview and course evaluation. Although this electronic meeting room is very 
conducive to providing each participant with unobstructed views of other participants, the facilitator, and the public 
screen, the same software can also be successfully used in an ordinary computerized classroom such as found on 
most college campuses. The software allows everyone to “speak all at once” via the computer. Students type in 
ideas at the same time, and each person sees the input of others, stimulating further thought. 




Figure 1. The Round Table Room with Collaborative Technologies 



Most collaborative software programs offer as the initial tool an Agenda program which provides a 
framework for the assessment or evaluation activities. It prompted us as facilitators to develop a specific plan, 
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specify the exact information needed, and keep track of all participants. Electronic Brainstorming is an idea- 
generating tool that enables students to share their ideas anonymously and simultaneously in response either to the 
specific questions posed to them, or to actually be given the chance to suggest additional questions they find 
important but missing. Tools such as Topic Commenter enable students to easily comment on the questions planned 
in advance by the facilitator of the assessment or evaluation session. The Categorizer program assists in analyzing 
the current information, sorting it into categories, identifying and adding missing ideas, and allowing for more in- 
depth comments on current ideas. Voting tools of most collaborative software allow participants to rate and rank the 
issues presented to them. Most collaborative systems also allow for online surveys. They store the data so that it can 
be automatically arranged into formatted reports, thus allowing for easy access of historical data. Three of these 
tools are explored further. 

The actual screens seen by students during a course evaluation session are shown in Figure 2. Three major 
questions were posed: “What helps you learn in this course?” “What do you think needs improvement?” and “What 
are your specific suggestions for change?” The electronic discussion is anonymous, with the software adding 
random numbers to student comments. This facilitates reacting to or answering another comment on the page. In our 
experience, we have found that professors elect to use this evaluation at midterm so that student perceptions can be 
incorporated into the planning for the rest of the course. 



GroupSystems - Summative - [Small Group Instructional Diagnosis (Topic Commenter)] 



^ File Edit Commenter Group Options Window Help 
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What helps you learn in this course? 
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What helps you learn in this course? 


3/0 


What do you think needs improvement? 


• 


1 liked the study groups that were required. {#7} 






4/0 




• 


The study guide in the website for the course was really helpful. 
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• 


Professor was always available during office hours. {#1 0} 


• 


Give more warning for major exams. Dont • 








big test on Friday. 
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1 felt like 1 could ask questions when 1 didnt understand 








something. {#11} 
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1 really Iked the fact that the professor didnt stand up and "talk at 
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Keep the computer lab open longer hours. 




us” all the time. We had to read the text and then we did things 








in class to make it more understandable. {#1 5} 
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Figure 2. Topic Commenter tool 
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One of the most useful tools in a collaborative system is one that lets students categorize the lists of ideas 
or answers that have been entered. In Figure 3, students were asked, “What helps you learn in this course?” The 
categories into which the answers might fall are shown at the right. This tool is mo st important in eliciting actual 
questions that students wish to pose. With a prompt such as, “What ideas do you wish to pose for discussion of 
departmental advising?” the facilitator can sort the questions into like categories, and transfer them to the discussion 
tools shown in Figure 2. All this is done in a matter of seconds. 

Another favorite tool for evaluation and assessment sessions is the one that allows students to make 
decisions and determine degrees of consensus or conflict. Multiple voting methods are allowed, including Yes/No 
and True/False, Top “n” favorites, customizable point scales, and Likert scales. Figure 4 shows a vote with only 4 
options, but many different scales are possible. Facilitators have the option to allow an odd number of choices, thus 
giving the student the alternative of “middle ground”. Results are immediately available, so students can see if their 
opinions vary a great deal from their peers. Viewing and discussing the results of the vote often leads to further 
questions and revealing comments. 
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GroupSystems - Summative - [Main Points - What Helps You Learn? (Categorizer)] 
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1. What helps you learn in this course? 

2. 1 liked the study groups that were required. 

3. The study guide in the website for the course was really helpful. 

4. Professor was always available during office hours. 

5. 1 felt like I could ask questions when I didn't understand something. 

6. 1 really Iked the fact that the professor didn't stand up and "talk at us" all the time. We had 
to read the text and then we did things in class to make it more understandable. 

7. You could tell that she really liked the subject. She got me excited about it too. 

8. The textbook was good, but the research articles were better and helped me learn more 

about the research process. j 
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Figure 3. Categorizer Tool 
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Figure 4. Voting Tool Rating Computer Lab Support 



One of the assessment activities that was most effectively supported with collaborative tools was the Senior 
Exit Interview. The purpose had always been to give students the opportunity to provide feedback to the department 
by providing them the chance to assess us on several dimensions. The Exit Interview includes questions on their 
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perceived academic accomplishments, our advising expertise, and the general support offered by the department 
during their time in college. Figure 5 shows the electronic discussion prepared for the group meeting of senior 
majors in Elementary Education. Information gained from the analysis of Exit Interviews is used to revise the 
curriculum, the teaching methodologies, the advising, and to make it even more accommodating to student needs. 
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Figure 5: Topic Commenter Tool with Senior Exit Interview 
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Assessing the Assessment Tools 

Classroom evaluation and student assessment just became easier and actually enjoyable by implementing 
the activities through collaborative technologies. Instead of listening to individual students in dozens of hours of 
individual meetings in exit interviews, and instead of endless meetings of faculty where opinions are lost once the 
meeting is over, we now have all the data that was gathered by the software stored in an organized fashion. It also 
became possible to store many years of data, which makes it easily accessible by institutional decision makers. An 
added benefit was the visibility of program improvement efforts to the college community. Students, faculty and 
other college constituents expressed positive feelings about their inclusion in these efforts. 

We find that students are eager to express their opinions in this “E-Listening” environment, and they tell us 
that the setting of an anonymous electronic discussion is very freeing and inviting. Faculty members find it easy to 
access the files, whether stored recently or in years past. The choice of several different types of automatic report 
generation makes structuring the data remarkably easy. 

Another guiding principle of the AAHE Assessment Forum is that “assessment works best when it is 
ongoing, not episodic. Its power is cumulative.” (Astin) Assessment starts “with the questions of decision makers, 
involves them in the gathering and interpreting of data, and informs and helps guide continuous improvement.” 
Collaborative technologies provide the means to collect ongoing, relevant information from all groups of campus 
constituents for the purpose of institutional improvement. The environment helps illuminate questions people care 
about. 
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